Skip to content

Refactor FederatedCredentials e2e test to use stable repository with config-based disabling#2104

Open
Copilot wants to merge 21 commits intomainfrom
copilot/refactor-federated-credentials-test
Open

Refactor FederatedCredentials e2e test to use stable repository with config-based disabling#2104
Copilot wants to merge 21 commits intomainfrom
copilot/refactor-federated-credentials-test

Conversation

Copy link
Contributor

Copilot AI commented Jan 29, 2026

❔What, Why & How

Federated credentials require stable repository identity. Refactored the test to reuse e2e-bingmaps.appsource instead of deleting/recreating tmp-bingmaps.appsource on each run. Test is currently disabled via config file pending Azure resource migration.

Helper functions added to e2eTestHelper.psm1:

  • CleanupWorkflowRuns: Deletes all workflow runs in a repository (ensures clean tracking state)
  • ResetRepositoryToSource: Hard resets repository content to match source via git reset --hard + force push (preserves identity, resets content)
  • Both functions include comprehensive PowerShell comment-based help documentation

Test flow changes in scenarios/FederatedCredentials/runtest.ps1:

  • Renamed repository: tmp-bingmaps.appsourcee2e-bingmaps.appsource (avoids tmp* cleanup pattern)
  • Requires pre-existing repository with federated credentials configured (errors if missing)
  • Resets to bcsamples-bingmaps.appsource content before each run
  • Cleans all workflow runs before triggering new ones
  • Re-applies test settings (ghTokenWorkflowSecretName) after reset
  • Timestamp-based workflow tracking (waits for correct CI/CD run)

Workflow changes in .github/workflows/E2E.yaml:

  • Created e2eTests/disabled-scenarios.json config file to manage disabled scenarios (array format with scenario and reason in each object)
  • Loads disabled scenarios from config and filters them out of the test matrix with comprehensive logging
  • Logging output: Shows which scenarios are disabled, how many were filtered, each disabled scenario with its reason, and final list of scenarios to run
  • Disabled scenarios do not appear in workflow runs (no jobs created)
  • Config-based approach allows easy management without modifying workflow code

Config file format (e2eTests/disabled-scenarios.json):

[
    {
        "scenario": "FederatedCredentials",
        "reason": "Azure resource migration work in progress"
    }
]

Technical details:

  • Mandatory parameters on helper functions
  • GUID-based temp paths to avoid conflicts
  • LASTEXITCODE checks after git operations
  • Removed unsupported -- --quiet flag from gh repo clone
  • Config-driven filtering prevents disabled scenarios from being included in matrix
  • Comprehensive logging provides transparency into filtering process
  • Disabled scenarios don't create jobs or consume CI/CD resources
  • Clean separation between workflow logic and disabled scenario management

To re-enable FederatedCredentials test: Remove the FederatedCredentials object from the array in e2eTests/disabled-scenarios.json.

When re-enabled, the test will use the stable repository instead of recreating it, maintaining federated credential authentication.

✅ Checklist

  • Add tests (E2E, unit tests)
  • Update RELEASENOTES.md
  • Update documentation (e.g. for new settings or scenarios)
  • Add telemetry
Original prompt

Refactor the FederatedCredentials end-to-end test scenario in microsoft/AL-Go so it reuses an existing temporary repository instead of deleting and recreating it on every run. The current approach creates a new repository called tmp-bingmaps.appsource copied from microsoft/bcsamples-bingmaps.appsource. Federated credentials no longer work with repository name-based matching, so the repository must be stable across runs.

Context:

  • Failing job URL: https://github.com/microsoft/AL-Go/actions/runs/21470716621/job/61842467415
  • Failure occurs when the test triggers the "Update AL-Go System Files" workflow in the newly created repository.
  • The test script is at e2eTests/scenarios/FederatedCredentials/runtest.ps1 and currently:
    • checks if the temp repo exists
    • deletes it if it exists
    • recreates it from template
    • then runs Update AL-Go System Files

Required changes:

  1. Modify e2eTests/scenarios/FederatedCredentials/runtest.ps1 to reuse the repo if it already exists.
    • Do not delete the repo anymore.
    • If the repo exists, reset it to match the source repo (microsoft/bcsamples-bingmaps.appsource) in a deterministic way.
    • Ensure the repo ends up with the same default branch content expected by the scenario (main).
  2. Add cleanup logic for workflow runs in the reused temp repo to prevent the run list from growing and to ensure the subsequent WaitWorkflow logic targets the correct run.
    • Before starting the scenario, delete older workflow runs (keep a small number like the last 5-10), or alternatively capture the run id of the workflow triggered and wait specifically for that.
  3. Ensure secrets and settings required for the test are still present/updated.
    • The script sets Azure_Credentials; preserve this.
    • The script passes addRepoSettings for ghTokenWorkflowSecretName; ensure this still works if repo already exists.
  4. Update any helper functions or supporting scripts if needed to support resetting repo contents and cleaning workflow runs.

Acceptance criteria:

  • The test no longer relies on deleting/recreating tmp-bingmaps.appsource.
  • The temp repo is stable across runs and compatible with federated credentials.
  • The script reliably waits for and inspects the workflow run it triggered, not an older run.
  • Add clear comments explaining why the repo is reused.

Relevant files:

  • e2eTests/scenarios/FederatedCredentials/runtest.ps1
  • e2eTests/Workflows/RunUpdateAlGoSystemFiles.ps1 (workflow trigger helper)
  • Any relevant helper module under e2eTests/e2eTestHelper.psm1 if needed.

This pull request was created from Copilot chat.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot AI and others added 3 commits January 29, 2026 10:51
…repository

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
…rove pagination and timestamp comparison

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI changed the title [WIP] Refactor FederatedCredentials test to reuse temporary repository Refactor FederatedCredentials test to reuse temporary repository Jan 29, 2026
Copilot AI requested a review from mazhelez January 29, 2026 10:59
…federated credentials

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI changed the title Refactor FederatedCredentials test to reuse temporary repository Refactor FederatedCredentials test to require pre-existing repository with federated credentials Jan 29, 2026
Copilot AI requested a review from mazhelez January 29, 2026 12:03
Copy link
Collaborator

@mazhelez mazhelez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work!

@copilot the following script is used in a workflow to clean up temp repositories: "Internal\Scripts\RemoveTempRepos.ps1"
Can you adjust the script or rename the repository, so that it doesn't get deleted?

…function, exclude tmp-bingmaps from cleanup

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jan 29, 2026

Good work!

@copilot the following script is used in a workflow to clean up temp repositories: "Internal\Scripts\RemoveTempRepos.ps1"...

Done. Updated Internal/Scripts/RemoveTempRepos.ps1 to exclude tmp-bingmaps.appsource from automatic deletion. The repository will now remain stable across cleanup operations. Commit: 2309402

Copilot AI requested a review from mazhelez January 29, 2026 12:24
…ce to avoid auto-cleanup

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI changed the title Refactor FederatedCredentials test to require pre-existing repository with federated credentials Refactor FederatedCredentials test to use stable e2e-bingmaps.appsource repository Jan 29, 2026
Copilot AI requested a review from mazhelez January 29, 2026 12:29
Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI requested a review from mazhelez February 5, 2026 08:21
…reason in each object

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI requested a review from mazhelez February 5, 2026 08:25
…ontext not available at job level)

Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Co-authored-by: mazhelez <43066499+mazhelez@users.noreply.github.com>
Copilot AI requested a review from mazhelez February 5, 2026 11:01
@mazhelez mazhelez marked this pull request as ready for review February 5, 2026 12:19
@mazhelez mazhelez requested a review from a team as a code owner February 5, 2026 12:19
Copilot AI review requested due to automatic review settings February 5, 2026 12:19
@mazhelez mazhelez enabled auto-merge (squash) February 5, 2026 12:20
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the FederatedCredentials end-to-end test to maintain a stable repository identity, which is required for federated credential authentication that no longer supports repository name-based matching. The test now reuses a pre-existing repository (e2e-bingmaps.appsource) instead of deleting and recreating it on each run. Additionally, a config-based mechanism for disabling scenarios is introduced.

Changes:

  • Renamed test repository from tmp-bingmaps.appsource to e2e-bingmaps.appsource and requires manual pre-creation with federated credentials configured
  • Added helper functions CleanupWorkflowRuns and ResetRepositoryToSource to maintain deterministic test state
  • Implemented config-based scenario disabling via e2eTests/disabled-scenarios.json with comprehensive workflow logging

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File Description
e2eTests/scenarios/FederatedCredentials/runtest.ps1 Refactored to reuse stable repository, reset to source state, clean workflow runs, and use timestamp-based workflow tracking
e2eTests/e2eTestHelper.psm1 Added CleanupWorkflowRuns and ResetRepositoryToSource helper functions with PowerShell comment-based documentation
e2eTests/disabled-scenarios.json New config file defining disabled scenarios with reasons (FederatedCredentials currently disabled)
.github/workflows/E2E.yaml Loads disabled scenarios from config and filters them from test matrix with comprehensive logging

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +345 to +373
function CleanupWorkflowRuns {
Param(
[Parameter(Mandatory = $true)]
[string] $repository
)

Write-Host -ForegroundColor Yellow "`nCleaning up workflow runs in $repository"

RefreshToken -repository $repository

# Get all workflow runs
$runs = invoke-gh api "/repos/$repository/actions/runs?per_page=100" -silent -returnValue | ConvertFrom-Json

if ($runs.workflow_runs.Count -eq 0) {
Write-Host "No workflow runs found"
return
}

Write-Host "Deleting $($runs.workflow_runs.Count) workflow runs..."
foreach ($run in $runs.workflow_runs) {
try {
Write-Host "Deleting run $($run.id) ($($run.name) - $($run.status))"
invoke-gh api /repos/$repository/actions/runs/$($run.id) --method DELETE -silent | Out-Null
}
catch {
Write-Host "Warning: Failed to delete run $($run.id): $_"
}
}
Write-Host "Cleanup completed"
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function only retrieves the first 100 workflow runs (per_page=100) and doesn't handle pagination. If a repository has more than 100 workflow runs, some will not be cleaned up. Consider adding pagination logic to iterate through all pages, or document this limitation in the function's documentation.

Copilot uses AI. Check for mistakes.
Comment on lines +127 to +132
$updateCreatedAt = [DateTime]$updateRun.created_at
$runs = invoke-gh api /repos/$repository/actions/runs -silent -returnValue | ConvertFrom-Json
$run = $runs.workflow_runs | Select-Object -First 1

# Find the CI/CD workflow run that started after the update workflow was created
$run = $runs.workflow_runs | Where-Object {
$_.event -eq 'push' -and [DateTime]$_.created_at -gt $updateCreatedAt
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using created_at from the update workflow run to find subsequent CI/CD runs may be imprecise. Since the Update AL-Go System Files workflow runs with -wait (line 117), it completes before returning. The CI/CD workflow is triggered by the push event when the update workflow completes, so comparing against the update workflow's created_at (when it started) rather than when it completed could cause issues if multiple workflows run concurrently. Consider using $updateRun.updated_at or adding a small time buffer to ensure we capture the CI/CD run triggered by the update workflow's completion.

Copilot uses AI. Check for mistakes.
Comment on lines +241 to +246
$beforeFilter = $filteredScenarios.Count
$filteredScenarios = $filteredScenarios | Where-Object { $disabledScenarios -notcontains $_ }
$afterFilter = $filteredScenarios.Count
if ($beforeFilter -ne $afterFilter) {
Write-Host "Filtered out $($beforeFilter - $afterFilter) disabled scenario(s)"
$disabledScenariosConfig | Where-Object { $filteredScenarios -notcontains $_.scenario } | ForEach-Object {
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic for logging filtered scenarios may show incorrect results. The condition $filteredScenarios -notcontains $_.scenario will match all disabled scenarios from the config that are not in the final filtered list, but this includes disabled scenarios that were never in the list to begin with (e.g., if they didn't match the scenariosFilter). To accurately show only the scenarios that were actually filtered out, you should capture the pre-filter list and check: ($scenariosBeforeDisabledFilter -contains $_.scenario) -and ($filteredScenarios -notcontains $_.scenario)

Suggested change
$beforeFilter = $filteredScenarios.Count
$filteredScenarios = $filteredScenarios | Where-Object { $disabledScenarios -notcontains $_ }
$afterFilter = $filteredScenarios.Count
if ($beforeFilter -ne $afterFilter) {
Write-Host "Filtered out $($beforeFilter - $afterFilter) disabled scenario(s)"
$disabledScenariosConfig | Where-Object { $filteredScenarios -notcontains $_.scenario } | ForEach-Object {
$scenariosBeforeDisabledFilter = $filteredScenarios
$beforeFilter = $filteredScenarios.Count
$filteredScenarios = $filteredScenarios | Where-Object { $disabledScenarios -notcontains $_ }
$afterFilter = $filteredScenarios.Count
if ($beforeFilter -ne $afterFilter) {
Write-Host "Filtered out $($beforeFilter - $afterFilter) disabled scenario(s)"
$disabledScenariosConfig | Where-Object { ($scenariosBeforeDisabledFilter -contains $_.scenario) -and ($filteredScenarios -notcontains $_.scenario) } | ForEach-Object {

Copilot uses AI. Check for mistakes.
# The CI/CD workflow should fail because the version number of the app in thie repository is lower than the version number in AppSource
# The CI/CD workflow should fail because the version number of the app in the repository is lower than the version number in AppSource
# Reason being that major.minor from the original bcsamples-bingmaps.appsource is the same and the build number in the newly created repository is lower than the one in AppSource
# This error is expected we will grab the version number from AppSource, add one to revision number (by switching to versioningstrategy 3 in the tmp repo) and use it in the next run
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The comment references "tmp repo" but the repository has been renamed to "e2e-bingmaps.appsource". This should be updated to reflect the new naming.

Suggested change
# This error is expected we will grab the version number from AppSource, add one to revision number (by switching to versioningstrategy 3 in the tmp repo) and use it in the next run
# This error is expected we will grab the version number from AppSource, add one to revision number (by switching to versioningstrategy 3 in the e2e-bingmaps.appsource repo) and use it in the next run

Copilot uses AI. Check for mistakes.
Comment on lines +235 to +236
# Load disabled scenarios from config file
$disabledScenariosConfig = Get-Content -path (Join-Path $ENV:GITHUB_WORKSPACE "e2eTests/disabled-scenarios.json") -encoding UTF8 -raw | ConvertFrom-Json
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code doesn't handle the case where the disabled-scenarios.json file doesn't exist. If the file is missing, the Get-Content command will throw an error and fail the workflow. Consider adding error handling or a check for file existence, or document that the file must always exist (even if empty as []).

Suggested change
# Load disabled scenarios from config file
$disabledScenariosConfig = Get-Content -path (Join-Path $ENV:GITHUB_WORKSPACE "e2eTests/disabled-scenarios.json") -encoding UTF8 -raw | ConvertFrom-Json
# Load disabled scenarios from config file (optional)
$disabledScenariosConfigPath = Join-Path $ENV:GITHUB_WORKSPACE "e2eTests/disabled-scenarios.json"
$disabledScenariosConfig = @()
if (Test-Path -Path $disabledScenariosConfigPath) {
$disabledScenariosContent = Get-Content -Path $disabledScenariosConfigPath -Encoding UTF8 -Raw
if (-not [string]::IsNullOrWhiteSpace($disabledScenariosContent)) {
$disabledScenariosConfig = $disabledScenariosContent | ConvertFrom-Json
}
}
else {
Write-Host "No disabled-scenarios.json found; proceeding with all scenarios enabled."
}

Copilot uses AI. Check for mistakes.
Comment on lines +136 to +138
# Fallback to the first workflow run if we can't find one based on timestamp
Write-Host "Warning: Could not find CI/CD run based on timestamp, using first run"
$run = $runs.workflow_runs | Select-Object -First 1
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fallback logic uses Select-Object -First 1 on all workflow runs without filtering by event type. This could select a workflow run that's not a push event (e.g., a workflow_dispatch event), which wouldn't be the CI/CD run triggered by the Update AL-Go System Files workflow. Consider filtering by event type in the fallback: $run = $runs.workflow_runs | Where-Object { $_.event -eq 'push' } | Select-Object -First 1

Suggested change
# Fallback to the first workflow run if we can't find one based on timestamp
Write-Host "Warning: Could not find CI/CD run based on timestamp, using first run"
$run = $runs.workflow_runs | Select-Object -First 1
# Fallback to the first push workflow run if we can't find one based on timestamp
Write-Host "Warning: Could not find CI/CD run based on timestamp, using first push run"
$run = $runs.workflow_runs | Where-Object { $_.event -eq 'push' } | Select-Object -First 1

Copilot uses AI. Check for mistakes.
Comment on lines 72 to 75
gh api repos/$repository --method HEAD
if ($LASTEXITCODE -eq 0) {
Write-Host "Repository $repository already exists. Deleting it."
gh repo delete $repository --yes | Out-Host
Start-Sleep -Seconds 30
if ($LASTEXITCODE -ne 0) {
throw "Repository $repository does not exist. The repository must be created manually with federated credentials configured before running this test."
}
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using gh api directly instead of invoke-gh is inconsistent with the rest of the codebase. The invoke-gh wrapper provides consistent error handling and authentication setup. This direct call should be replaced with invoke-gh wrapped in a try-catch block to handle the case where the repository doesn't exist.

Copilot uses AI. Check for mistakes.
Comment on lines +419 to +452
if ($LASTEXITCODE -ne 0) {
throw "Failed to clone repository $repository"
}

# Fetch the source repository content
Write-Host "Fetching source repository $sourceRepository..."
invoke-git remote add source "https://github.com/$sourceRepository.git"
if ($LASTEXITCODE -ne 0) {
throw "Failed to add remote source for $sourceRepository"
}

invoke-git fetch source $branch --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to fetch branch $branch from source $sourceRepository"
}

# Reset the current branch to match the source
Write-Host "Resetting $branch to match source/$branch..."
invoke-git checkout $branch --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to checkout branch $branch"
}

invoke-git reset --hard "source/$branch" --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to reset branch $branch to source/$branch"
}

# Force push to update the repository
Write-Host "Force pushing changes..."
invoke-git push origin $branch --force --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to push changes to $repository"
}
Copy link

Copilot AI Feb 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The $LASTEXITCODE checks after invoke-gh and invoke-git calls are unnecessary and incorrect. These wrapper functions use cmdDo, which throws an exception when a command returns a non-zero exit code (see Actions/Github-Helper.psm1 line 296-298). The exit code checks will never be reached because an exception will be thrown first. These checks can be removed, as the try-catch block will handle errors.

Suggested change
if ($LASTEXITCODE -ne 0) {
throw "Failed to clone repository $repository"
}
# Fetch the source repository content
Write-Host "Fetching source repository $sourceRepository..."
invoke-git remote add source "https://github.com/$sourceRepository.git"
if ($LASTEXITCODE -ne 0) {
throw "Failed to add remote source for $sourceRepository"
}
invoke-git fetch source $branch --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to fetch branch $branch from source $sourceRepository"
}
# Reset the current branch to match the source
Write-Host "Resetting $branch to match source/$branch..."
invoke-git checkout $branch --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to checkout branch $branch"
}
invoke-git reset --hard "source/$branch" --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to reset branch $branch to source/$branch"
}
# Force push to update the repository
Write-Host "Force pushing changes..."
invoke-git push origin $branch --force --quiet
if ($LASTEXITCODE -ne 0) {
throw "Failed to push changes to $repository"
}
# Fetch the source repository content
Write-Host "Fetching source repository $sourceRepository..."
invoke-git remote add source "https://github.com/$sourceRepository.git"
invoke-git fetch source $branch --quiet
# Reset the current branch to match the source
Write-Host "Resetting $branch to match source/$branch..."
invoke-git checkout $branch --quiet
invoke-git reset --hard "source/$branch" --quiet
# Force push to update the repository
Write-Host "Force pushing changes..."
invoke-git push origin $branch --force --quiet

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants